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a data link (4) that further includes an optical fiber controller (J) coupled to each data link (4). Each controller (1) is interconnect- 
ed through fiber (4) for high speed data transfers, from one set of nodes (6) to another. The controller (1) is capable of connection 
implementing three and four cable interfaces. The data is transmitted through the fiber (4) serially but the controller (1) is adapt- 
ed to receive parallel data and convert to serial form and vice versa. 



FOR THE FVBFOSBS OF INFORMATION ONLY 



Codes used to identify States party to the PCI* on the frunt of pamphlets publishing international 
applica^ons under the PCT. 



AT 


Aioiriu 


FR 


Kruiicc 


MR 


Mauritania 


AU 


AttslraJia 


GA 




MW 


Malawi 


BB 


Bartxidus 


CB 


Uottod Kmplom 


ML 


Nctfacrlaiub 


BE 


Belgium 


CH 


Guinea 


NO 


Norway 


BP 


Buf^ioa Rbo 


CR 


Greece 


NZ 


New Zcalotul 


BC 


Buffi^iria 


HU 


llunfary 


PL 


Foloml 


ai 


Bcmin 


le 


Inslaod 


PT 


PUrU^I 
kumiinia 


BR 


Bra/K 


rr 


Uafy 


RO 


CA 


Canaila 


jp 


JafKin 


RO 


Rubian RxleniUon- 


CF 


Cluniral Africaii Rvpublic 


KP 


Duniocratic Feople'a Kirpublk 


SD 


SuUatt 


CC 






oTKonai 


SE 


Swvdctt 


CH 


Swiixcriand 


KR 


Republic oT Korea 


SK 


Skivak Republic 


O 


€^3lc itlvtnris 


KZ 


KasnLhslan 


SN 


Scttegal 


Ch% 




IJ 


licclitciKilein 


SU 


Soviet UrtKXl • 


CS 




IK 


Sri toaLi 


m 


ClKiil 


cz 


Ctveli Rcputilk 


f.U 


loixctnbourg 


TC 


Togo 


DC 


(Jcrmuii) 


MC 


Monaco- 


UA 


Ukraine 


DK 


OcnfiKtrk. 


MC 




US 


Uniuat Slalo oT America 


ES 


Spain 


Ml 


MiiU 


VN 


Vict Nam 


n 


Httbml 


MN 


Mbiwolia 







wo 93/19422 



PCT/US93/fi2fB9 



1 

FIBER OPTIC MEMORY COUPLING SYSTEM 

FXBtP Of THig lHVByT?:P^ 

This invention relates to a novel fiber optic aenory 
interconnection for linking special memory busses of processing 
nodes and has particular application for real time data processing 
systems operating at large distances. 

BACKGROUWD OF THE INVEWTTQW 

Systems for updating memories in coupled nodes are known from 
U.S. Patent No. 4,991,079 the content of which is here incorporated 
by reference and from U.S. Serial Ho. 07/403,779 filed September 8, 
1989, which is a continuation of Serial No. oe/880,222 filed June 
30, 1986, now abandoned, the content of which is here incorporated 
by reference; all of which are commonly owned with this present 
application. Such systems use two ported memories and are used to 
transfer writes to one memory in one node automatically and at high 
speed to memory in other nodes with the intervention of a CPU. 
Such systems, however, have a distance limitation of about 120 feet 
and eight nodes. The present invention is an improvement that 
enables such systems to be connected over a distance. The present 
state of the art allows for connections of 3 kilometers and up to 
ten kilometers with a high speed data interface. 

SUMMARY OF THE. TWVEWTTQW. 

The present invention provides a means for connecting such 
memory coupled processing systems over a large distance and 
provides for high speed data transfers between the systems, copying 
data from the memory of a node in one system to the memory of a 
node in another system. 

Other and further advantages of the present invention will 
become readily evident from the following description of a 
preferred embodiment when taken in conjunction with the appended 
drawings. 
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2 BRIEF DESCRIPTION OP THE DRAWINGS 

3 Figure 1 illustrates the system of the present invention; 

4 ri9ure 2 illustrates the fiber to memory coupled 

5 system controller (FMC) ; 

6 Figure 3 illustrates the FMC to memory coupling bus interface; 

7 Figure 4 illustrates the method of handling a memory vrite 

8 transfer by the FMC; 

9 Figure 5 illustrates an example of address translation as 

10 perfoinned by the present invention; 

11 Figure S illustrates the FKC separated into data path 

12 quadrants; 

13 Figure 7 illustrates Quadrant 0 of the FMC; 

14 Figure 8 illustrates Quadrant 1 of the FHC; 

is Figure 9 illustrates Quadremt 2 of the FMC; { 

16 Figure 10 illustrates Quadrant 3 of the FMC; 

17 Figure 11 illustrates a parallel/serial latch used .in the 

18 pr€tseht invention; 

19 Figure 12 illustrates the basic packet format; 

20 Figure 13 illustrates the format for a data packet; 

21 Figure 14 illustrates the format for a general purpose async 

22 data pa^et; 

23 Figure 15 illustrates the format for a data packet of a HCS-II 

24 jniultidrop console; 

25 Figure 16 illustrates the format for an interrupt packet; 

26 Figure 17 illustrates the mode of operation of the FUC for the 

27 handling of async serial data; 

28 Figure 18 illustrates the mode of operation for handling 

29 special async data pass through; 

30 Figure 19 illustrates handling of interrupts by the FMC; ^ 

31 Figure 20 illustrates an example of network linking clusters 

32 of nodes; 

33 Figure 21 illustrates a secondary backup high speed link for 

34 the system; 

35 Figure 22 illustrates internal lobpback for memory vrite 

36 transfers; ] 
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Figure 


3 

23 illustrates external loopback for memory write 


3 


transfers; 




4 


Figure 


24 illustrates memory coupling bus loopback; 


5 


Figure 


25a illustrates intelmal loopback for general purpose 


is 


asyhc data; 




7 


Figure 


25b illustrates external loopback for general purpose 


8 


async data; 
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Figure 


25c illustrates port-port loopback; 


10 


Figure 


26 illustrates cluster FMC error handling; 


11 


Figure 


27 illustrates hub FMC error handling; 


12 


Figure 


28 illustrates typical configurations of the present 


13 


invention; 




14 


Figure 


29 illustrates a star configtiration of the present 


15 


invention. 





16 

17 DETAILED DESCRIPTiOy OF THE PREFERRED EMBODIMENT 

18 This invention relates to a high speed data interface system 

19 using fiber optic memory interconnection as shown in Figure 1. 

20 This interface system connects memory coupled systems over 

21 distances up to 10 kilometers* Each memory coupled system 

22 comprises a data link or memory coupling bus 5 to which up to eight 

23 nodes 6 are coupled each comprising a processor, I/O, a two or more 

24 ported memory and a processor to memory bus* Write/read sense 
"25 controllers couple the busses and the memory so writes only are 

26 sensed and reflected to the memories of other nodes without CPP 

27 intervention. This is described in detail in the patent and 

28 application noted above, both of which are here incorporated by 

29 reference.. The bus 5 of each memory coupled system is connected to 
, 30 a fiber- to-memory coupling system controller CFMC) 1, Each FMC has 

31 both an input and output port for connection with another FMC. The 

^ 32 input port 2 is for receiving transmitted data from another memory 

33 coupled system and the output port 3 is for transmitting data to 

34 another memory coupled system. The transmission of the data is 

35 through fiber optic cables 4. 

36 The Fiber Optic Memory Coupling System (FOMCS) also provides 
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2 the capability for nodes on separate MC busses to exchange async 

3 serial data across the fiber link and for a node on one HC bus to 

4 interrupt a node on another. Eac& FKC supports 8 general purpose 

5 async ports and 16 interrupt lines (8 input and 8 output). The 

6 multidrop console link incorporated in the MCS-il fourth ceible is 

7 also supported. These features are intended to allow remote 

8 booting of nodes across the fiber and to provide a means of 

9 . synchronizing the actions of nodes. 

10 Throughout this document, the term "MCS cluster" (or sin^^ly 

11 "cluster") is used to refer to a MC bus and its attached nodes and 

12 FMC. In configurations of only two or three clusters, pairs of 

13 FttCs are used to directly connect HC busses (see Figure 28) • In a 

14 configuration of more than three clusters, a FOKCS hub is used to 

15 connect all the clusters in a star configuration (see Figure 29). 
1€ The configuration programming link shown in Figures 28 and 29 

17 is used to establish the operational mode of each FHC. FMCs which 

18 arfe hot directly connected to the programming link receive 

19 progt^unmihg information via packets sent over the fiber link. 'The. 

20 Ftic 1 can be seen in more detail in Figure 2. The FMC includes a 

21 Receive Data Path 7, an output latch 9, a receive FIFO 11, a 

22 receive Error Detection Circuit 13, two receive latches 15 and 17, 

23 a receiver 19, an input latch 10, a hit and translation RAH 12, a - 

24 transinit FIFO 14, a transmit Error Detection Circuit 16, two 

25 transmit latches 18 and 20 and a transmitter 22. 

2^ The FHC 1 consists of four main sections: the memory coupled 

27 system 5 interface, the data paths (Rx and Tx) 7 and 8, the high 

28 speed serial data link interface 4, and the microprocessor. These 

29 ar^s are delineated viking dashed lines in the functional block 

30 diagram (Figure 2). The Fiber Transition Module (FTM) connects to 

31 the FMC high speed serial link interface. The FTM is a )cnown and 

32 conventional piece of hardware and serves to connect the electrical 

33 signals to corresponding light signals and vice versa. 

34 The Memory Coupling (HC) bus interface is the FHC's link to 

35 the Memory Coupling System (MCS) and the nodes on that bus. The 

36 FMC implements a three-cable, 90-signal interface for those HCS 
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2 networks that use a standard HC bus (24-bit addresses)^ and a 

2 four-cable, 120 -signal interface for those MCS networks that use 

4 the 28-bit addresses provided in MCS-XX. Provision is also made in 

6 the four-cable interface for eventual support of 32-bit addresses. 

6 The FMC supports all the address lines defined in the four-cable 

7 interface; however, in. an environment where addresses are a full 32 

8 bits, the FMC will only reflect into and out of the first 256 

9 megabytes of memory. 

10 The FMC appears as a typical MCS node on the bus, using one 

11 of the nine bus IDs <0 - 8) available in MCS or MCS-II. The MCS 

12 used by the FMC is set during initialization by the configuration 

13 programming link, 

14 Details 6f the FMC interface to the HC bus are illustrated in 
1 ) 15 Figure 3* The FMC MC bus interface receives bus transfers by 

16 . latching the memory address, memory data, flag bits and parity on . 

17 the rising edge of DATA VAtID (assuming DATA VALID is not being 

18 driven by the FMC itself) . The two different bit counts for the 

19 address, flags and parity lines reflect differences between the 
2Q three cable MCd bus and the four cable MCS-II bus. The smaller 

21 . counts apply to the three cable bus. Note that 32 bits of address 

22 are indicated for the four cable MCS-II bus rather than 28. As 

23 mentioned earlier, signals to support the additional four address 

24 bitis are reserved in the fourth cable for future expansion. One 
.25 bit of odd parity is provided for each byte of address and data, 

26 resulting in the seven or eight parity bits shown in Figure 3. 

27 The flag bits (|ualify the received transfer. For memory 

28 write transfers, one flag bit (referred to as the *>F-bit") 

29 indicates whether the memory write is to a byte location in memory 

30 as opposed to a half word or word. The other two flag bits, which 
. 31 are only present in the MCS-II bus, differentiate between memory 

32 write transfers and other types of transfers. While only memory 

33 write transfers appear on an MCS-II bus in a cluster, the MCS-II 

34 bus in the FOHCS hub is used to distribute interrupt, async and 

35 other types of data in addition to memory write traffic. 

^ 36 As indicated in Figure 3, the address, flags and parity are 
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2 treated as 32, 3 and 8 bit quantities, respectively, when received 

3 by the FlIC. If a three cable MCS bus is connected to the FMC, the 

4 receivers for the eight most significant address bits, the tvo 

5 extra flag bits and the eighth parity bit are disabled. Zeros are 

6 put in the MC input latch for the missing addreiss and flag bits and 

7 a one is inserted for the missing parity bit. In a four cable 
B environment, 32 bit addresses are received but the FMC will discard 
9 any received memory write transfer in which the most significant 

10 four address bits are not zero. 

^2. If receipt of a transfer causes the FMC Tx FIFO 14 to become 

12 half full, the FMC will drive GLOBAL BUSY on the MC bus to prevent 

13 overflow. GLOBAL BUSY is deasserted when the Tx FIFO 14 becomes 

14 less than half full. The CONTROL BUSY signal is only present in 

15 the MCS-II bus and is only utilized in the FOMCS hub. If receipt 

16 of a non-memory-write type transfer causes the FMC FIFO used to 

17 hold such transfers to become half full, the FMC will drive CONTROL 

18 BUSY to prevent overflow. When the FIFO becomes less than half 

19 full, CONTROL BUSY is deasserted. 

20 Before the FMC can generate a transfer on the MC bus, it must 

21 first acquire the right to access the bus. To accomplish this, the 

22 FMC asserts ^e REQUEST line on the bus which corresponds to the 

23 MCS ID the FMC has been programmed to use. The FMC then monitors 

24 the corresponding GRANT line. When the bus arbiter asserts the 

25 GRANT line, the FMC drives the memory address, memory data^ flag 

26 bits and parity on the bus. The FMC's MCS ID is also driven on the 

27 node id lines of the bus. On the next rising edge of the VALID 

28 ENABLE signal, the FMC drives DATA VALID. (The VALID ENABLE signal 

29 is a free running clock generated by the bus arbiter to synchronize 

30 the actions of nodes on the bus,) 

^31 Note that the MC output latch illustrated in Figure 3 only 

32 supplies 28 bits of the memory address rather than 32. The driver 

J3 inputs for the remaining four address bits are tied to ground 

34 (i*e., the bits are forced to be zeros). 

35 receipt of a memory write transfer packet causes the FMC 

36 Rx FIFO 11 to become half full and MC bus burst request mode is 
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2 enabled, the FMC will drive BURST REQUEST. Asserting the BURST 

3 REQUEST signal ^ which 13 only present in the HCS-II bus, causes the 

4 bias arbiter to enter a mode in which only the FMC is granted access 

5 to the bus. After asserting BURST REQUEST and delaying long 

6 enough to guarantee propagation of the signal to the arbiter, the 

7 FHC deasserts GtOiSAL BUS^; If no other node on the bus is 

8 asserting GLOBAL BUSY, the arbiter will begin issuing grants to the 

9 FHC. If one or more other nodes is asserting GLOBAL BUSY» the 
id arbiter waits for the l>usy condition to clear and then begins 

11 issuing grants to the FMC, Note that the FKC can safely deagsert 

12 GLOBAL Bt^SY even if it is unable to accept more transfers from the 

13 bus because the arbiter will only grant bus access ^o the FHC. 

14 When the FHC has unloaded enough packets front its Rx FIFO 11 
. 15 to cauise the fill level to drop below half full, BURST REQUEST is 

is deaisserted and the arbiter returns to the normal "fairheBs" 

17 arbitration scheme « If the FMC is unable to accept more transfers 

is from the bus, it will assert GLOBAL BUSY before deasserting BURST 

19 REQUEST. 

26 If the FMC's Tx FIFO is less than half full, the FMC will 

21 keep BURST REQUEST asserted for 8 MC bus cycles and then release it 

22 for 16 cycles and thien assert it for 8,. release it for 16 and so on 

23 until the Rx FIFO fill level drops below half fiill. If both the 

24 Rx and TX FIFOs are half or more full, l^e FMC will assert BURST 

25 REQUEST continuously until enough packets have been unloaded from 

26 the RX FIFO to cause the fill level to drop below half full* Once 
.27 BURST R&QUEST is deasserted, the arbiter returns to the normal 

28 **faimess" arbitration scheme* If the FMC is unable to accept more 

^ 29 transfers from the bus, it will assert GLOBAL BUSY before 

4 3d deasserting BURST REQUEST. 

31 . Support for burst request mode in the FMC is enabled or 

^ 32 disabled via a configuration programming coimnand. Only one FMC on 

33 a given MC bus can have burst request mode enabled but at least one 

34 FMC in every FMC-to-FMC link must have the mode enabled to ens?' fe 

35 reliable operation.; Note that enabling burst request mode in tu^ 

36 FMC only means that the FMC is able to drive BURST REQUEST on the 
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2 bus. For burst request mode to be useful, tlie four cable MCS-^II 

at environment with Memory Coupling Controllers (MCC's) as the bus 

4 arbiter/terminators is required. MCC's provide bus termination and 

5 arbitration in a conventional manner as known from the previously 
is noted patent and application and are sometimes referred to as 
7 reflective membxry controlleris. 

6 The FMC Rx and Tx data paths 7 and 8 move diata between the 
9 MCS interface and the high speed serial data link. Ihey also 

10 interface to the microprocessor allowing asynchronous data, 

11 interrupt transfers and flow control infonaation to move between 

12 clusters. 

13 Figure 4 illustrates the manner in «iiich the FMC processes 

14 MCS memory write transfers. As indicated^ not all transfers are 

15 transmitted as packets over the high speed serial data link. Only 

16 those transfers which correspond to memory writes into selected 

17 regions of memory are transmitted over the high speed link. 

18 Transfers which do not fall within these selected regions are 

19 simply discarded. 

20 In the other direction, packets containing memory write 

21 transfers are received over the high speed link and decoded by the 

22 FMC. Note that in the Rx path 7 there is no hit/translation ItlM so 

23 all memory write packets received over the high ^eed link cause 

24 memory write tremsfers to be geniarated on the MC bus. Kheh a 

25 packet has been received which represents a write into memory, the 

26 FMC requests the use of the MC bus and when the request is granted, 

27 generates a msaory write transfer on the bus. 

28 In the Tx path 8, the regions of memory to be reflected are 

29 defined during configuration prograiuming of the IMC. During 
3d programming, the total memory address space is viewed as a sequence 

31 of 8K byte {2K word) blocks. Multiple regions as small as 8K bytes 

32 can be defined. Any niimber of regions, segregated or concatenated, 

33 up through the entire address range may be established. 

34 Mother feature of the FMC which is illustrated in Figure 4 

35 is memory address translation. Prior to packetizihg a memory 

36 write transfer and transmitting it over the high speed link, the 
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memory addriess is modified • HCS physical addresses are 24 bits 
long (MCS-I) or 28 bits long (MCS-IX) • A MC5 physical address is 
translated by using the most significant 11 bits (MCS-I) or 15 bits 
(MCS-II) to address a hit/translation RAM on the FMC. The value 
read from the RAM is the most significant 15 bits of the new 
address. 

Note that the least significant 13 bits of a memory address 
are unaffected by the translation process* ' Xhus, 9)c byte, blocks, 
are mapped from the addreiss space of the source MC hus to that of 
the destination MC bus and vice verse. As illustrated in Figure 5, 
this feature is very useful because it allows clusters to share 
memory regions vhicb reside in the different places in each 
cluster's physical address space. 

The contents of the hit/translation RAM are established 
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1 during configuration programning of the FHC« The size of neooiy 

2 addresses (i^^e.^ 24 or 28 bits) oh the source HC bus is also 

3 estia>li8hed during configuration programming* Ea^ location in the 

4 hit/translation RAN represents a 8K byte bloeJc of toiemory ' and 

5 contains a hit bit and the 15 bit translation value. If the hit 

6 bit is setir memory writes into the 8K byte memory block are 

7 refli^etctd over the high speed link. If the bit is reset, such 

8 meinory writies ar^ ignored. 

9 In the content of the KCS star niatworl: configuration, the FHC 

16 region selection and address translation features can be U6ed to 
il. segregaite the NCS clusters connected to the hub into groups such 

12 that memory write traffic is only reflected within a group, not 

13 between groups. Note that if an administrative system (i.e., not 

14 one of the nodes in the star network) is used to program the FMCs, 

15 a secure networ)c can be achieved in which cluster groups can 
IS coexist but not affect each others meniory. If total isolation of 

17 groups is not desired, overlapping regions can be used. 

18 ' Tb^ FHC microprocessor has the ability to interject data into 

19 ^ and remove data from both the Tx and Rx data paths 7 and 8. Such 

20 data is referred to as control data to distinguish it from memory 

21 write transfers. General purpose async data, HCS-II multidrop 

22 console async data and interrupt pulses are all treated as control 

23 data. 

24 fttien the FHC receives data over the general purpose async 

25 ports, it fonss the data into paclcets and injects those packets 

26 into the transmit data stream sent over the high speed link. When 

27 a packet of async data is received by the FHC, the bytes are broken 

28 out of the packet aiid sent over the appropriate general purpose 

29 async port. HCS-II multidrop console data is handled in a similar « 

30 fashion. 

31 Khen an enabled input interrupt line is pulsed, the FHC ^ 
3t2 generates a packet that is sent over the high speed link. Upon 

33 receipt of an interrupt packet from the serial data link, the FHC 

34 pulses the appropriate output interrupt line. 

35 Other control transfers include configuration programming 
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1 information, high availability nessages/ error indications, and 

2 reset indications, 

3 Flow control in a MCS network is accomplished by means of 

4 £ V control bits in the packets sent from FNC to FNC and by means 

5 o£ MC bus busy signals. Internal to the FMC, memory write data 

6 transfers and other types of data transfers (i.e., async, 

7 interrupt, multidrop, etc*) are handled separately so that flow 

8 control may be applied to non*miemory wite transfers without 

9 affecting the miemory Write traffic. Each packet sent from PMC to 

10 FMC contains two flow control bits, one to cause the raceiving FMC 

11 to cease or resume transmission of memory write transfer packets 

12 and another to cause it to cease or resume transmission of other 

13 types of packets* This notion of two separate data streams also 

14 applies to the hub MC bus where there are separate bus busy 

15 signals, one for memory writ« transfers and another for all other 

16 types of transfers. 

17 Ah additional type of flow control is burst request mode. 

18 Burst request mode Is n^scessary to ensure that lock ups do not 

19 occur on FMC-to-FMC links. When link utilization is high in both 

20 directions, the potential exists for a FMC-*to-FMC link to lock up 

21 in a condition ih %^ich both PMCs are asserting busy on their 

22 respective NC busses. Because the busy condition on the HC busses 

23 prevents the PMCs from generating bus transfers, the FMCs are 

24 unable to reduce tba fill levals of their tx FIFOs 14 and will 

25 therefore never deassert bus busy. 

26 Burst request mode alleviates this problem by allowing a FMC 

27 to \mload transfers from its Rx FtFO 11 while ensuring that the FMC 

28 will not have to accept additional transfers into its Tx FIFO 14. 

29 This means the FMC can accept more memory write transfer packets 

30 from the remote FMC which in turn allows the remote FMC to unload 
* 31 its Tx FIFO 14 and eventually clear the busy condition on the 

32 remote NCS bus. Clearing the busy condition allows the remote FMC 

33 to unload its Kx FIFO 11. The remote FMC can then accept more 

34 memory write transfer packets which allows the local FMC to unload 

35 its Tx FIFO 14 And clear the busy condition on the local MC bus. 
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1 As shown in Figure 6, the FMC data paths are logically 

2 subdivided into quadrants. In quadrant 0, MC bus transfers are 

3 received and moved to the Tx FIFO 14. The Tx FIFO 14 serves as the 

4 boundary between quadrants 0 and 2. -In quadrant 2, transfers 

5 removed from the Tx FIFO 14 are packetized and transmitted over the 

6 high speed serial link 4. 

7 Packets are received over the high speed link 4 in quadrant 

8 3 and the packet contents are moved to the Rx FIFO ll. The Rx 
. 9 FIFO 11 serves as the boundary between quadrants 3 and 1. in 

lb (quadrant 1, information removed from the Rx FIFO 11 is used to 

11 generate transfers on the MC bus* 

12 An important feature of the FMC design, particularly from a 

13 diagnostic perspective, is that the latches in each quadrant can 

14 be accessed in both a parallel and serial fashion. During normal 

15 operation of the FMC, data moves through the latches using the 

16 parallel interface. The alternate serial interface allows the 

17 microprocessor on the FMC to shift data serially into and out of 

18 the latches. 

19 Figure 7 shoVs a detailed view of quadrant 0. Processing 

20 begins in quadrant 0 when the control logic 25 detects that the MC 

21 input latch 10 has been loaded. The input latch is unloaded and 

22 odd parity is computed on the address and data. The computed 

23 pari^ is then compared to the received parity. At the same time, 

24 the most significant address four bits are checked to see if any 

25 of them are non*zero. 

2« While the parity and address checks are taking place, 15 bits 

27 of the address are used to address the hit/translation RAM 12. (If 

28 the address came from a three cable MCS bus, the upper foiir of the 

29 15 bits used to address the RAM will be zeros.) The least * 

30 significant 15 bits of the 16 bit value read from the RAM |»ecome 

31 the new, or translated, address bits. The most significant bit 

32 of the value read from RAM is the window hit bit. 

33 The destination of the MC transfer is determined by the 

34 parity and address checks, the hit bit, and the flag bits received 

35 With the transfer. If any of the most significant four address bits 
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1 Is non-zero/ the transfer xs discarded. If the received parity 

2 does not equal the computed parity, the transfer is clocked into 

3 the toicroprocessor interface FIFO 21 » The transfer is also placed 

4 into the micro FIFO if the parity is good but the flag bits 

5 indicate a control type traxisfer (which 6nly bcctirs in the MCS 
• 6 hub) • The destination of a aemory write type transfer with good 

7 parity depends on the setting of the hit bit. If the hit bit is 

8 reset, the transfer is si&ply discarded. If the hit bit is set, 

9 the memory data^ translated memory address and flag bits are 
10 clocked into the Tx FIFO 14. 

XI The contents of the hit/translation lUOf 12 are initialized 

12 by t)ie FMti microprocessor. To change a location in the RAM 12 , the 

13 microprocessor first puts the new value into the loading , buffers 

14 24. The micrppzrocessor then causes GLOBAL BUSY to be asserted On 
is the MC hOB and waits long enough for any transfer in proglress' to 

16 pass through the MC input latch 10. Then, the microprocessor 

17 causes the hit/translation UAH 12 to be written with the value from 

18 the loading buffers. GLOBAL BUSY is subsequently deasserted. 

X9 Figiire 8 shows a detailed view of quadrant 1. Quadrant 1 

20 processinig begins when the control logic 25 detects that the MC 

21 output laitch 9 is empty and either the Rx FIFO 11 is not empty or 

22 the micro interface 21 is requesting use of the latch 9. If there 
2i3 is something in the FIFO 11 and the micro interface 21 is not 

24 reqtuesting, the FIFO 11 is read and parity is computed on the 

25 memory address and data. The data, address, flagis and computed 

26 parity are then clocked into the MC output latch. 

27 The micro intezlace 21 is only used when a FHC in a FOHCS hub 
^ 28 needs to distribute interrupt, async of other control information 

29 to the other FMCs in the hub. If control busy is not asserted on 

30 the hub bus, the micro interface logic 21 requests use of the MC 

31 output latch 9. When the latch 9 is available, the control data 

32 is serially shifted into the latch 9. Odd parity for the transfer 

33 is computed by the microprocessor and shifted into the latch 9 

34 following the data* 

35 Figure 9 shows a detailed view of quadrant 2. Processing 
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1 begins in quadrant 2 when the control logic detects that the Tx 

2 latches 18 and 20 are empty and finds that the Tx FIFO 14 is not 

3 empty or that the microprocessor has loaded the micro interface 

4 latch. If the micro latch has been loaded, the contents of the 

5 latch are transferred to the Tac latches 18 and 20. Of the 72 bits 

6 tramsf erred from the' miicro latch, 64 are also clocked Into a pair 

7 of Ebc (error detection code) generators 16a and i6b, 19ie 

8 generated eight bit EiDC aixid the 72 bits from the micro latch are 

9 clocked into the tx latches 18 and 20. The contents of the Tx 

10 latches 18 and 20 are then passed to the high speed link serial 

11 tremsmitter 22 in two 40 bit transfers, 

12 If the micro latch is empty but the Tx FIFO 14 is hot, the 

13 memory data, address and flags are read from the FIFO 14. An 

14 additional nine- bits are generated by the control logic to iiiake a 

15 total of 72. Again, an EDO is generated on 64 of the 72 and an 80 

16 bit packet is clocked into the Tx latches 18 and 20. The contents 

17 of the Tx latches 18 and 20 are then passed to the transmitter 22. 

18 If the control logic detects that either the Rx FliFO ll or the 
id micro FIFO in quadrant 3 are half full, flow control f lag 6its are 

20 . asserted in the 80 bit packet clocked into the Tx latches 18 and 

21 20. If no packet is availeible to clock into the latches, the 

22 control logic 25 causes a special f lov control packet to be clocked 

23 into the Tx latches 18 and 20 for transmission to the remote FMC. 

24 Separate flov control bits are defined for tiie Rx FIFO 11 and the 

25 quadrant 3 micro FIFO so that the flov of memory write ud non 

26 memory write packets can be throttled separately, once the FIFO(s) 

27 become less than half full, a packet is sent to inform the remote 

28 FUC that it can resume transmission of one or both types of 

29 packets. ' 

30 Figure 10 shows a detailed view of quadrant 3. Quadremt 3 

31 processing begins when the first 40 bits of a packet are received ^ 

32 over the high speed serial link. Control lc7ic 25 causes the 40 

33 bits to be clocked into a staging latch 15. When the rest of the 

34 packet arrives, the entire 80 bit packet is clocked into the Rx 

35 latch 17. From the Rx latch 17, 64 bits of the packet are moved 



W093/W422 



PCf/US93/02839 



15 

to a pair of EDC generators/cheOcers 13a and 13b • The received EDC 
is compared to that generated and an indication of the outcome is 
provided to the control logic 

The destination of the contents of the packet depend on the 
packet type bits examined by the control logic and the result of 
the EbC check. If the £i)C check fails# the packet contents are 
moved to the micro interface FIFO 23* The packet contents are also 
moved to this micro FIFO 23 if the packet typa flags indicate that 
the packet contains async, interrupt or other control information, 
otherwise, the memory address, data and flags are moved to the Rx 
FIFO 11. 

Among the 12 packet bits examined by the control logic 25 are 
the flow control bits, one for throttling memory write padket 
transmission and the other for throttling non memory write packet 
transmission. A set flow control bit in a received packet causes 
the FMC to cease transmission of the corresponding type of packet 
until a packet is received with that flow control bit reset • 

Figure 11 shows a block diagram of the AMD 29818 latch used 
to implement the MC input and output latches 9 and 10 and the Rx 
latches 15 and 17 and Tx latches 18 and 20 on the FMC. In the FMC 
design, the normal data path through these latches is the parallel 
one from the D input through the pipeline reg^ister 29 to the q' 
output* The alternate serial path is accessible by the FHC 
microprocessor and is primarily used for diagnoetic purposes. Note 
that the mux 3b (irtiich is controlled by the MODE signal) allows 
either the contents of ^e shadow register 31 or the D input to 
serve as the input for the pipeline register 29. Also, note that 
the output of the pipeline register 29 is feed back into the shadow 
register 31. aSlus, with appropriate hardware control, data can 
enter the latch in serial and exit in parallel or enter parallel 
and exit in serial. 

These features are exploited by the FMC design which allows 
the microprocessor to serially load a latch in one quadrant, move 
the data along the normal parallel path from that latch to a latch 
in another quadrant and then serially read the contents of the 



W693/19422 PCr/US93/fl2839 



16 

1 destination latch. By comparing the data loaded into the first 

2 latch with that read bac)c from the second, the microprocessor can 

3 determine if the data path is functional. 

4 In the fOHCS architecture, the high speed serial link is 

5 implemented with a transmit/receive pair of Gazella Hot Riod c^ps. 

6 The Gazelle Hot Rod trauismitter chip converts 40-bit parallel data 

7 into serial data that is transmitted over a fiber or coax link. 

8 At the remote end of the link, tha data Is recbnverted to the 

9 original parallel data by a Hot Rod receiver chip» Over this link, 

10 the FHCs exchange information in the form of 80*bit packets The 

11 Gazelle transmitter sends each 80-bit packet as two 4d-bit data 

12 frames. When no packet is available to send, the transmitter sends 

13 sync frames to maintain link synchronization. 

14 The data oh the serial data link is 4B/5B NRZI (Non-Return 

15 to Zero, Invert on ones) encoded. Thiis allows ttm receiver to 

16 recover both data and clock signals from the daita stream, 

17 precluding the need for a separate clock signal to be sent along 

18 with the data. Data rates as high as 1 6bps (one billion bits per 

19 second) are supported. 

20 Transmission error detection is accomplished via ah eight 

21 bit error detection code (EDO) which is included in each packet. 

22 Of the 72 remaining bits in the packet^ only 64 are included in the 

23 EDO calculation. The eight unprotected bits are the firist four 

24 bits of each of the 40-bit halves of the packet. 

25 The basic packet format is shown in Figure 12. 

26 Bits 0 and 40 are used to distinguish the two 40-*bit 

27 sections of the packet. When receiving a packet, a FHC always 

28 expects the first bit of the first 40-bit section to be a zero and 

29 the first bit of the second 40-bit section to be a one. 

ao The BSD and BSC bits (bits 1 and 41, respectively) are used 

31 by the transmitting FHC to throttle transmissions by the IHC 

32 receiving the packet. The BSD bit is used to tell the receiving 

33 FHC to cease or resume transmission of memory write data. packets; 

34 the BSC bit is used to tell the receiving FHC to cease or resume 

35 transmission of control packets. The bits are set to tell the 
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1 receiving FHC to cease the associated type of transmissions; they 

2 are reset to tell the receiving FMC to resume transmissions. 

3 The TYP bit (bit 4) indicates the packet type* If the bit 

4 is reset, the packet contains a memory Write transfer. If the bit 

5 is set, the packet is designated as a control packet. Control 

6 packets are used to transfer async data, interriiipit data and any 

7 other type of information which FMCs must exchange. 

8 The VAL bit (bit 5) indicaltes whether the cross^-hatched 

9 portions of the packet (bits 6-39 and 44 71) actually contain 

10 any valid information. If the VAL bit Is set, the receiving F8C 

11 will consider the information in the croiss^hatched portions to be 

12 valid. If the VAL bit is reset, the receiving FMC will only pay 
( j 13 attention to the flow control information encoded in the packet 

r4 (i.e., the settings of the BSD and BSC bits). When a FHC does not 

X% have any data to send, it will periodically transmit paelcets with 

16 the VAL bit reset and the. flow control bits set or reset as 

17 appropriate. 

18 As mentioned earlier, bits 0-3 and bits 40 * 43 are not 

19 included in the EDO calculation. This means that the BSD bit 

20 ahd/6r the BiSC bit may be in error in a received packet and ho EDC 

21 error will be detected. However, even if an inViailid flow control 

22 indication is received and acted upon, the next packet received 

23 will almost certainly correct the problem. 

24 The shaded portions of the packet (bits 2 3 and bits 42 ^ 

25 43) are reserved. The convention of shading reserved portions of 

26 packets is followed thrdughout this documents 

27 Bits 72 -79 Contain the EDC which is computed 6h bits 4 -39 

28 and bits 44 * 71. The EDC is an 8-bit check byte generated 

29 according to a modified Hamming Code which will allow detection 

30 of all single- and double-'bit errors and some triple-bit^ errors. 
* 31 Because of the MRZI encoding scheme used by the Gazelle Hot ftod 

32 Chips, noise on the high speed serial link medium will cause 

33 sequential do\ible-bit errors which the receiving FMC will be able 
Y 34 to detect by comparing the EDC in the packet to that computed 

35 during receipt of the packet. 
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1 The memory write transfer packet is used by the FMC to 

2 transmit a memory address and associated data over the high speed 

3 link, the address and data represent a memory write transfer which 

4 the FHC received from the KC bus. The format of the packet is 

5 shown in Figure 13. 

6 the PBT bit (bit 7) is the F^bit associated with the memory 

7 write. The F-bit is set if the memory write transfer represents 

8 a write to a single byte of memory. The F*bit is res^t if the 

9 ' trandfer representiB a write to a half*-word or word. 

16 Kdte that if the memory address is a 24-bit address, bits 44- 

11 47 of the packet will contain zeros. 

3^2 The general purpose async data packet format is shown in 

13 Figure 14. 

X4 The packet cacn contain up to five bytes of async data. The 

15 first four bytes are placed in bitiB 8^-39 with the first byte in 

16 bits 8-15, the second in 16-23 and so on. The fifth byte is placed 

17 in bits 64-71. The receiving FMC determines the actual number of 

18 data bytes in the packet by checking' the Byte Count field (bits 45- 

19 47)i 

20 The Destination Cluster field (bits 48 - 55) contains the 

21 address of the HCS cluster for whic^ the packet is ultimately 

22 intended. The FMC in the destination cltESter uses the Destination 

23 Port field (bits 56 - 63) to determine which general purpose async 

24 pdrt to use when transmitting the async data, decoded from the 

25 packet. • - 

26 The BSK bit (bit 44) indicates that the originating FMC 

27 received an async break indication. When the BRK bit is set, the 

28 packet does not contain any async data (i.e., the Byte Couht field 

29 contains a zero) . Upon receipt of a general purpose async packet 

30 with TOK set, the FMC in the destination cluster flushes its output 

31 buffer for the destination port and generates a break on that port^. 

32 The MCS-II multidrop console data packet format is illustrated 

33 in Figure 15. 

34 The packet can contain up to four bytes of async data. The 

35 four bytes are placed in bits 8-39 with the first byte in bits 



i 

W093/W422 



PCr/USM/02839 



19 

1 8 «^ iSf the second in 1.6 ^ 23 and so on* The receiving FKC 

2 determines the actual nuaber of data bytes in the packet by 

3 checking the Byte Count field (bits 45 - 47)* 

4 The Destination Cluster field (bits 48 - 55) contains the 

5 addreiss of the KCS cl'uster for which the packet is ultimately 

6 intended « The contentii of this field are only seaningful if the 

7 BRD bit is reset « 

8 The Source Cluster field (bits 56 - 63) contains the address 

9 of the cluster in which the originating FMC resides. Because 

10 multiple packiets are required to send a multidrop message, a FMC 

11 in a cluster may be receiving parte of messages f rom two or more 

12 clusters at the same time. The source cluster field in the packets 

13 allows the receiving FMC to segregate message pieces based on 
'14 source cluster and thus properly reconstruct ttie messages* 

15 The SdM bit <bit 64 > indicates whether the async data in ^e 

16 packet represents the start of a multidrop message or not* If the 

17 SOM bit is set, the packet contains the first 1-4 bytes 

18 (depending on the Byte Coimt) of a multidrop message. If the SOM 

19 bit is reset, the packet contains data from within the body of a 

20 . mejssage. 

21 The EOM bit (bit 65) indicates Whether the asyhc data in the 

22 packet represents the end of a multidrop message or not. If the 

23 EOM bit is set, the packet contains the last 1-4 bytes (depending 

24 dh the Byte Count) of a multidrop message » If the £0M bit is 

25 reset, at least one more pacl^t of data from the mesfsage will 

26 follow. 

27 The BRD bit (bit 67) is set if the packet is to be broadcast 

28 throughout the FOMCS network to all clusters. If the BRD bit is 

29 set, all FMCs in a FOMCS hub will forward the packet to their 

30 reepective remote clusters. All cluster FMCs receiving such a 

31 pabket will accept it (assuming they have multidrop support 

32 enabled) . 

33 The interrupt packet format is shown in Figure 16. 

34 The Destination Cluster field (bits 48 - 55) contains the 

35 address of the MCS cluster for which the packet id ultimately 
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ixiteiided^ T&e FKC in the diestination clustet uses th^ Destination 
Line field (bits 56 «^ 63) to determine vhich ou^ut interrupt line 
to pulse. 

A Motorola 68000 microprocessor on the FMC provides the 
configuration system interface as well as data management for all 
transfers other than memory vrite transfers. This includes all 
aisynchronous data, intetrrupts and error logging. It also provides 
diagnostic capabilities driven through the configuration 
programming interface. 

Four Signetics 66681 DDliRTs on the FMC provide support for 
the exchanges of async serial data between the nodes and/6r devices 
in separate kcs clusters. The async support is designed in such 
a way that communicating entities need not be aware that the async 
data is actually traveling over a high-speed fiber or connection, 
irroa the perspective of the entitles,, dnmunication is n^t 
functionally different from the case where the nodes and/or devices 
are directly connected via R5-232 cables. Ko special protocol 
inf orimation has to be inserted by the communicating entities into 
the async data st3*eam to aaiow the data to move through the MCS 
networSc. Figure 17 illustrates the handling of async serial data 
by a FHC in a cluster- 
As eisync serial data is received from a node or device, the 
FMC packetizes the data and transmits the packets over the high 
speed serial liiik. In addition to async data, «ach packet contains 
an address whlc^ facilitates routing of the packet throu^ the KCS 
network. Zn the other direction, the FKC receives and decodes 
packets of async data from the high speed link. The address in the 
packet is discarded and the async data is passed to the node or 
device. 

The address in an async packet is designed to allow routing 
of the packdt through, a MCS hub. To accomplish this, the address 
consists of two fields, a destination cluster field and a 
destination port field. The destination cluster field indicates 
the cluster to which the packet is to be* routed. The destination 
port field indicates which async link of the FMC in the destination 
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1 Cluster that the data is intended for. the destination FHC 

2 validates the address when it receives the packet. If the 

3 destination cluster address does not match the local cluster 

4 address, the packet is discarded. 

$ The FMC maintains a list of eight asyne addresses, one for 

6 each general purpose asyhc port. When async data is received from 

7 one of the eight ports, the FHC looks up the asyhc address for that 

8 port and includes it in the packet (s) sent over the high speed 
. .9 link. Thus, there is a static connection between each general 

10 purpose async port of a FMC and an async port of a FMC in some 

11 other cluster. 

12 The contents of the list of async addresses and the local 

13 cluster address are established during configuration programming 

14 of the FMC« The physical characteristics of the async links (i«e», 
1^ baud rate, character leiigthr etc.) are also established during 

16 configuration pirogrammihg. 

17 In a MCS hub, the FHCs are programmed to operate in a special 

18 async data pass-through mode. This mode of operation is 

19 illustrated in Figure 18* Async packets received over the high 

20 speed link are decoded by the receiving FMC and sent over the hub 

21 MC bus to the other FMCs in the hub. The address field inserted 

22 by the FHC at the originating cluster is passed along over the bub 

23 Md bus as well as the async data. Each FHC in the hiib which 

24 receives the address end async data from the MC bus compares the 

25 destination cluster field to the address of the cluster at the 

26 remote end of ite high speed link. If a match occurs, the FMC 

27 builds and transmits an async packet over the high speed data link. 

28 The remote cluster address used by a FMC in the hub to perform 

29 the routing function just mentioned is established during 

30 configuration programming. Since the async links of a FMC in a hub 

31 are not connected and are ignored by the FMC, no async address list 

32 or async link physical' characteristics can be programmed. 

33 To accommodate the case where the high speed link of a FMC in 

34 one hub is directly connected to a FMC in another hub (rather than 

35 to a MCS cluster) , async address checking may also be disabled in 
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1 a FHC via configuration programming^ Thus, all asynp packet 

2 traffic over the bus of one hub will also appear on the bus of the 

3 other hub* 

4 The general purpose async ports of a FHC support hardware flov 

5 control using DTR and CTS. When a EMC wishes to transmit over a 

6 general pitirpose asyhc port and hardwetre flow control is enabled 

7 for that port, the FHC only transits as Idhg as it sees CTS 

8 asserted. A loss of CTS during transmission causes the port to 

9 . cease tran»ission until CTS reappears « Thus, the devicie connected 
ID to ah fMC async port can throttle IKC data transmission for that 

11 port by asserting and deasserting CTS. 

12 HxB FMC can also throttle a device which is sending it data 

13 over a general purpose async port assuming that an appropriate ( 

14 cable is used and the device pays attention to the CTS signal. 

15 When the FKC determines that its receive buffer for the port is 
lis nearly full, it deasserts DTR to inform the device connected to the 

17 port of the condition^ Assuming a cable which coxinects the port*s 

18 DTR signal to the device's CTS signal, the device will see a loss 

19 of CtS» This should cause the device to cease tranisiitission until 
20' the FKC asserts DTR to indicate that more data cain be accepted. 

21 During configuration programming of the FMC, async flow 

22 control using DTR and CTS can be enabled or disabled on a per port 

23 basis. If DTR/CTS flow control is not desired, FMC general purpose 

24 async ports can be configured for XOK/XOFF flow control or flow 

25 control can be disabled altogether. 

26 To deal tirith the situation in which a FMC is connected to an 

27 async data source which is transntitting data much faster than the 

28 device connected to the destination FMC can accept it, an async 

29 flow conttol mechanism is also ii^lemented between FKCs. An fMC 
3d can send flow control packets to throttle the transmission of asyhc 

31 data packets by the FMC connected to the source of the async data. i 

32 Flow control can be asserted on a per async port basis so that the 

33 flow of async packets for other async ports is unaffected. 

34 The FHC supports 16 interrupt lines: eight input lines and | 

35 eight output lines. The lines allow an interrupt pulse generated 
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1 by a node or device in one MCS cluster to be passed, in effect, 

2 across the MCS netvorX; to a node or device in another cluster. The 

3 pulse enters the FHC in the originating cluster via an input 

4 interrupt line, is passed across thie MCS network as a packet and 

5 leaves the FMC in the destination cluster via an output interrupt 

6 line. As With async data, this is designed in such a way that the 

7 node ne^d not be awiare that the interrupt pulse is travelling over 

8 a high speed serial link and not actually directly connected. 

9 An oviarview of interrupt passing for a FOMCS configuration is 

10 showh in Figuria 19. The process involved in passing an intierrupt 

11 across FOMCS is very similar to that used to pass async data. The 

12 FHC detects the pulse on the input interrupt line and constructs 

13 a special ihterxtipt packet which is transmitted oyer the high speed 
: 14 link. The interrupt packet contains an address which facilitates 

15 routing through the FOM^ network. At the destination FkC, the 

16 packet is decoded and the address is used to determine which 

17 interrupt line should be pulsed. 

18 As in the async case, the address in an interrupt packet 

19 consists of tw6 fields. The destination cluster field identifies 

20 the cluster to which the packet is to be routed. The destination 

21 lin^ field indicates which output interrupt line is to be pulsed. 

22 As with async packets, the destination FMC validates the cluster 

23 field of a received interrupt packet and discards the packet if the 

24 destination cluster does not match the local cluster address. 

25 The FMC mttintains a list of eight interaupt ^ine addresses, 
2€ one for each input interrupt line. When the FMC detects a pulse 

27 on one of the input interrupt lines, it looks up the line address 

28 and includes it in the packet sent over the high speed link. Thus, 

29 thsre is a static connection between each input interrupt line of 

30 a FHC and an output line of a VMC in some cither cluster. The 

31 contents of the list of intetrupt line addresses are established 

32 during configuration programming of the FMC. 

33 In a FOMCS hub, the FHCs are programed to operate in 

34 pass-through mode very similar to that described earlier for async 

35 data. Interrupt packets received over high speed link are 
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1 distributed over the MC bus to the other FMCs in the hub* Each FMC 

2 in the hub receives the interrupt line address from the KC bus and 

3 coinperes the cluster field to the address of the cluster at the 

4 remote end of its high speed link. If a mat^ occurs ^ the FMC 

5 builds and transmits an interrupt packet over the high speed link. 

6 The remote cluster address used by a FKC in the hub to perform 

7 the routing function just mentioned is established during 

8 cohf iguzation programing. Interrupt address checking can also be 

9 disabled in a IMC to accommodate the situation where hubs are 

10 directly connected. Since the interrupt lines of a FMC in a hub 

11 are not connected and are ignored by the FMC, no input interrupt 

12 addr<^ss list can be ^ro^ammed. 

13 one channel of a Signetics 68681 DDART on the FMC is used to 

14 provide support for the MCS«*II multidrop console link. (The other 

15 channel of this DtJART is used for the configuration programming 

16 link.) TtiB FMC behaves basically as a gateway to allow console 

17 traffic to flow between nodes in separate MCS-II clusters* In each 

18 cluster, the FMC intercepts messaged intended for remote nodes and 

19 transmits them as packets to the FMCs in the appropriate clusters. 

20 • The FMCs receiving the packets decode them and send the messages 

21 to the destination nodes. The fact that this is happening is 

22 essentially tremsparent to the nodes. An example of a FOKCS 

23 network linking clusters of MCS-II nodes appears in Figure 20. 

2t4 In the MCS-II multidrop console environment, one of the drops 

25 acts as the master and all others are slaves, only the master can 

26 initiate message exchanges on the multidrop link; a slave can only 

27 transmit after receiving a poll message from the master. The FKC 
2iB in a cluster assumes that there is a local master in the cluster. 

29 When the FMC receives a message addressed to a remote node, it 

30 transmits the message as one or more pacA:ets over its high speed 

31 serial link; when the FMC receives one or more packets from the 

32 high speed link containing a message from a remote node, it buffers 

33 the message until it is polled by the master. Then, the FMC 

34 transmits the saved message over the local multidrop link. Kote 

35 that a cluster FMC*s slave address on the multidrop link is its KC 
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1 bus node id. 

2 The handlingi of multidrop message data in the POMCS network 

3 is basically the saine as that for general purpose async data and 

4 interrupts^ Hultidrop data is routed through the FOHCS netvork in 

5 special pac)cetd. Each aultidrop data packet contains a destination 
6 . cluster address (which is validated by the destination FMC) and 

7 part of a message. Unlike interrupt packets or packets of general 

8 purpose async data ^ multidrop message packets also contain a source 

9 cluster address. The receiving FMC uses the source address to 

10 segregate pieces 6>f messages from different clusters into different 

11 buffers* 

Xi The sbuirce' cluster address which a node inserts into a 

13 message is supplied by the FMC in the cluster. The multidrop 

14 master periodically sends a request-'Cluster-address message to the 

15 tAC which causes it to broadcast a response message containing the 

16 cluster address to all nodes in the cluster* This cluster, address 

17 is the address eisftablished during configuration programining via the 

18 - Define Cluster Address command. Mote that in a MCS-II cluster 

19 where there is no FMC to supply the source cluste. 

2D communication between nodes is unaffected because the source 

21 cluster address field is hot included in messages exchanged by 

22 nodes in the same cluster. 

23 In FOM^ configurations where a hub is present, multidrop 

24 data is routed from the receiving FMC to the other FMCs in the hub 

25 via the hub MC bus. Packets are then built and transmitted from 

26 FMCs ih the bub to the FMCs in the destination clusters. As with 

27 general purpose async and interrupts, a FMC in the hub looks at the 

28 destination cluster address to determine if it should build and 

29 transmit a packet over its high speed link. If the destination 

30 cluster matches the address of the cluster connected to the FMC*s 

31 serial data link, the FMC forwards the multidrop data. 

32 The MCS-II multidrop broadcast capability is also supported 

33 by FOMCS. When the FMC in a cluster receives a message over the 

34 local multidrop link that is to be broadpast to all nodes, it sends 

35 the message across its high speed link in packets which have a 
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1 broadcast flag set. FMCs in a hub will always f orvard such packets 

2 regardless of the contents of the destination cluster field. All 

3 cluster BKCs receiving the broadcast message will transmit it over 

4 their local multidi^op linlcs when polled by the local masters. 

5 During conf iguration programming, a cluster FMC is programmed 

6 with a list of remote cluster addresses* When multidrop support 
i is enabled, the FMd intercepts ahy mesisage on the miatidrop link 

8 which is addressed to a node in one of the clusters in the list. 

9 The local inultidrop master ca(h obtain the list frcim the FMC by 

10 seiidifig it am inquiry message.. The FHC responds by encoding the 

11 list into a remote-duster-list message which it sends over the 

12 amltidrbp link. 

13 The FKC supports an optional configuration which allows ( 

14 automsitic f Clover to a secondaury high speed serial link should the ^ 

15 primary link go down. this high availability feature is 

16 illiistrated in Figure 21. 

17 The secondary FMCs in each cluster monitor the health of the 

18 primary FMCs. Each secondary FKC is configured for the same MC bus 

19 node id as the primary it monitors but the seicondaries do not 

20 interact on the MC bus as long as the primaries are healthy. Ih 

21 other words, all memory write traffic between the clusters goes 

22 over the primary high speed link until a failure occurs. Likewise, 

23 all other types of packet traffic (i.e., async, interrupt, etc.) 

24 go over the primary link until a failure is detectM. 

25 To keep the secondary FMCs informed as to their health, the 

26 primaries periodically send their respective secondaries an "I'm 

27 okay** indication via a health-check link. The UART of a Motorola 

28 68901 MFP on the FMC is used to implement the health-*check link. 

29 The primaries also periodically exchange test packets to detenine ^ 

30 if the primary link is still functioning. 

31 If a secondary FHC does not receive an "I*m okay** indication ^ 

32 within a specified time period since the last indication or if the 

33 secondary receives a link failure indication from the primary, the 

34 secondary initiates the fail^over process by: 1> forcing the | 

35 primary FMC to cease operation, and 2) sending a failure-detected 
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1 packet to the secondary in the renote cluster* The renbte 

2 secondary FMC then forces its primary FMC to cease operation, the 

3 remote secondary then sends a fail-over-^coniplete packet back to 

4 the secondary vhich detected the failure* Subsequent conmunication 

5 between the clusteris occuris over the secondary high sp(eed linkr 

i A secondary FMC forces a primary to cease operation by 

7 - asserting a special additional signal which is included in the 

8 health-check link cable connecting the primary and secondary*. When 

9 asserted this signal places the primary FMC into a 

10 hardware-controlled offline state. in this state, all I/O 

11 interfaces on the FMC are dis2a>led so that the FMC is imable to 

12 assert any signals^ oh the MC bus, transmit over the high speed 

13 link, transmit over emy of its asyhc links or pulse any of its 
- ,14 output interrupt lines. The effect is identical to that achieved 

15 by putting the FMC online/offline switch into the offline position. 

16 The (old) primary FMC is held in this offline state until the 

17 secondary FMC (which is the new primary) is reset via a power 

18 cycle, hardware reset or a received Reset command, 

X9 While the old primary FMC is being held in this offline state, 

20 it monitor^ the- heal th^^check link (if it is operational enough to 

21 do so) for the receipt of "I'm okay»» characters from the new 

22 primary. If the old primary receives such characters, it 

23 reconfigures itself to assumie the secondary role. Thus, when an 

24 attempt is made to xetum the old primary FMC to an online state, 

25 it behaves as a secondary and remains in a (fixnware-controlled) 

26 offline states This prevents the old primary from contending with 

27 ..the new primary on the MC bus. 

28 The fail-ovd^r process can also be initiated manually via 

29 the configuration programming linlc. To cause a fail-over, a 

30 command is sent over the programming link to one of the secondary 

31 FMCs. The fail-o^er proceeds as described ebove and when it is 

32 complete, a response indicating such is returned via the 

33 programming link. 

34 While the previous discussion relates to high availability 

35 of the high speed link between clusters, the high availability 
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1 featvire can be extended to the hub in a FdHCS star network. High 

2 availability in a hub is achieved by having a secondary FMC for 

3 eac^ prinary Fife in the hub. Thus, each cluster is connected to 

4 the hub by two high, speed links ^ a primary and a secondary. 

5 Fail-over to a secondary link is essentially the same as the 

6 cluster to cluster case. 

7 Note that to really achieve high availability in a cluster^ 

8 nodes which have async or interrupt connections to the primary 

9 FMC laust have a redundant set of eonheetions to the secondary. 

10 When the secondary FMC takes over, the nodes must switch over to 

11 the secondary async and interrupt connections. To facilitate this 

12 process, one of the output interrupt lines of a secondary FMC can 

13 be tteed to inform the node(s) that fail'-^er has occurred. During • 

14 configuration programming of a secondary FMC, an output interrupt ^ 

15 line can be designated for this piirpose. 

16 The fail-over process is not automatically reversible. 

17 Once the secondary FMCs have taken over, they become in effect the 

18 primaries. Nheii the previous primary FMCs and/or high speed link 

19 are repaired, they must be programmed to behave as the secondaries 
2a (if they have no already reconfigured themselves as secdhdaries) . 

21 Configuration of a FMC as a primary or secondary occurs 

22 during configuration programming. The maximxm time period that a 

23 secondary will allow between **I*m olcay" ihdications from the 

24 primary is also established. Ttk» high availability feature can 

25 also be disabled altogether* 

26 Another usage of the microprocessor is to allow diagnosis 

27 of an individual FMC and/or high spee.d serial link without 

28 bringing the entire MCS network offline » This capability is 

29 particularly useful in the star network environment where it is ' 

30 undesirable to shut down the entire hub just to diagnose a problem 

31 between the bub and a paurticular cluster. If the problem turns out ^ 

32 to be the high speed link or the FMC in the cluster, it can be 

33 corrected without taking down the entire hub. Of course, software 

34 resynchrtinization will probably be necessary between the affected ^ 

35 cluster and the other clusters it was communicating with but the 
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1 fest of the clusters connected to the hub can continue to 

2 connUnicate without interruption* 

3 special operational nodes, referred to as hub diagnostic and 

4 Cluster diagnostic addes, are available to allow the FMG to be 

5 isolatted from the high speed toiix or fiber link for testing 

6 jpurpdses. In thesie ioiodeis, the FkC operation is identical to normal 

7 hub 6r cluster mode except that the output of the Gazelle Hot Rod 

8 Tx chip is directly corinected to the input of the Rx chip. To 

9 configure the FMC to operate in hiib. of cluster diagnostic mode, the 

10 Set THC Onlihe/dffiine coinmaAd used to place the mc in a 

11 f irmWare--conttr611ed offline state* Then, the desired mode is 

12 8elecH:ed via the S^t C^erational Mode comm€md and the FkC is 

/ ) 13 returned to an online state by means of a second Set FMC 

. V ..V .. . • 

:14 Online/Offline command,. 

lis Individual functional areas of the FMC can be tested by 

16 sending Specify Diagnostic Lbopback Data commands to the TAc» 

17 Variations of the Specify Diagnostic Loopback Data command can be 

18 used to invoke memozy wite transfer loopback^ async data loopback 

19 and interrupt lbopback* A FMC will only accept sucb commands when 

20 - in the firmware-controlled offline state. 

21 The FMC suppbirts three dia^ostic loopback liiodes for memory 

22 write transfers: internal^ external and MC bus loopback. Internal 

23 loopback loops data through the FMC from the MC bus input latch to 

24 the MC bus output latch. External loopback tests the same path 

25 except that the data is actually transmitted over the lii^h speed 

26 serial link and looped back via an external loopback cable 

27 connection* intenial loopback mode is shown in Figure 22. 

28 External loopback ttode is shown in Figure 23. MC bus loopback 

29 loops data from the MC bus output latch to the MC bus input latch 

30 via the MC bus. This loopback mode is illustrated in Figure 24. 

^ 31 Note that when internal or external loopback is perfozmed, 

32 the FMC hit/translation JIAM ttiist aljso be programmed appropriately 

33 to achieve the desired effect. Loopback modes can be used to 
{ 34 specifically test reflection region or address translation logic 

35 or the hit/translation HAM can be programmed such that all regions 
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1 arc reflected arid address translation does riot actually do anything 

2 (i.e., the original address and the translated addriess are the 

3 saiaie) • 

4 in ihtemal loopbacic node, the KC bus interface of the FMC 

5 is disabled and the Gazelle Hot Rod chips are configiured such that 

6 the serial output of the transmitter chip is connected directly to 

7 the serial input of the receiver chip. The Specify Diagnostic 

8 Loopback Data comnand which initiated the loopbadc also contains 

9 m address and data pattern to be looped through the FKC hardware. 

10 The FKC inserts the address and data pattern into the HQ bus input 

11 latch. The address and data proceed through the Tx path and at 

12 the end of the Tx path are transmitted serially in a paclcet. The 

13 Packiat is looped back thtough the Rx path and the address and data 

14 erid iip in the MC bus output latdti. The FKC removes, the addriess and 

15 data from the latdi arid includes them in the response sent back 

16 via the prbgrainming link. 

17 In external loopback mode, the MC bus interface of the FHC 

18 is disabled. Again, the Specify Diagnostic Loopback Data command 

19 which initiated the loopback contains an address and data pattern 

20 to be looped through the FMC hardware and the external loopback 
'^X bonnectibn. The FMC inserts the address and data pattern into the 

22 MC bus input latch. The address and data proceed through the Tx 

23 path, axul at the enid of the TX path are transmitted serially in a 

24 packit* The packet is looped through the external loopback 

25 connection, rciceived again by the FHC and moves through the Rx path 

28 to the MC bus output latch. A response is sent back via the 
27 programming link containing the address and data pattern read from 
IB the latch. 

29 In MC bus loopback mode, the high speed serial link interface ^ 

30 is disabled amd the MC bus interface of the FMC is enabled but 

31 functions somewhat differently from normal. To perform the MC bus « 

32 loopback, the FMC inserts the address and data pattern from the 

33 Specify Diagnostic Loopback Data command into the MC bus output 

34 latch. TtiB FMC MC bus interface hardware. then requests the bus and 

35 when it receives a grant, generates a transfer on the MC bus with 
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th0 data valid signal d^assert^d. This causes any nodes on the bus 
to ignore the transfer. The FHC MC bus interface hardware, 
however, lis configured to receive its own transfer.. The looped 
data is read f rem the HC bus input latch aiid returned in the 
response sent back via the prograaiming link. 

The FMC supports three diagnostic loop^ack aodi&d for general 
pun>ose async data: internal, external, and port-port. The three 
varieties are illustrated in Figure 25. The FKG perforns the 
requested async loopback as the re^lt of a Specify Diagnostic 
Loopback Data ccomiiand ireceived over the conf iguration programing 
link. 

The Sjpecify Diagnostic Loopback Data conoaamd inforns the FMC 
ais tb which async port(8) will participate in the loopback test and 
provides the data„ to bia looped. If ah async port is selected for 
internal loopback, thie FMC initializes the port hardware to wrap 
tranismitte^ data back to the receive side of the port. The FMC 
then causes the port to transmit the data. if the port is 
functioning correctly , the FMC imnediately receives the data back 
again. The received data is passed back over the prograiiming link 
in the cdaisand response/ Zf no data is received, the FMC returns 
a response indicating such. 

External loopback requires that a loopback plug be connected 
to the port which wraps transmitted data back to the receive pin 
of the port. The FMC initialises the port hardware for normal 
operation and trahsmite the data via the specified jptort. Agaih, 
the data should be imnediately received from the same port. The 
response to the COTmand contains the received data or indicates 
failure of the loopback operation. 

Port-^port loopback requires that a IIS^232 cable connect the 
two selected ports* The FMC initialises the ports for normal 
operation and transmits the data via the port specified in the 
Specify Diagnostic Loopback Data command. Vhe response to the 
coBonand contains the data received from the other port or indicates 
failure of the loopback operation. 

The FMC supports extamal loopback for the interrupt lines. 
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1 A Specify Diagnostic Loopback Data coamtand to the FMC selects a 

2 pair of interrupt lines that will participate in the loopback, one 

3 input and one output. Note that loopback requires that a wire 

4 cdhnieet the selected interrupt lines. To perform the loopback, 

5 the FMC generates a pulse on the output line of the pair and 

6 reports in its command response whether or not a pulse vas detected 

7 on the input line. 

8 The nic si^ports.two diagnostic loopback modes for the MCS-Ii 

9 multidrop console link: internal and eaetemal. Internal loopback 

10 is performed exactly as with the general puri>08e async ports. 

11 External loopback is functionally identical to the method used with 

12 the general purpose async ports, but because of the unique nature 

13 of the multidrop link, no external loopback cable is required. f 

14 HOweveir, the external loopback test will drive the multidrop link. 

15 Therefore, if testing is desired without bringing the entire 

16 multidrop link offline, the MCS cable which carries the multidrop 

17 link sihoxad be unplugged from the FMC chassis backplane prior to 
la initiating dia^estie teists and replaced A^en testing is concluded. 

19 The FMC st^ports two diagnostic loopback modes for the high 

20 availability health-check links internal and external. Both modes 

21 are performed in the same manner as with the general purpose async 

22 ports. 

23 Each paekeit transferred across the high speed serial link in 

24 a FOMCS configuration contains an error detection code (EDC) which . 

25 allows the receiving FMC to determine if an error occurred during 

26 transmission. Packets received with good EDC arc also checked for 

27 illegal or illogical bit settings. Parity is checked for transfers 

28 received across the MC bus and transfers received with good parity 

29 are also decked for illegal or illogical bit settings. ' ■ ' ' 

30 For packet errors and transfer errors detected by the 

31 receiving FMC, the FOMCS philosophy is to report the error to the ^» 

32 node(s) in the nearest cluster rather th2ui reporting the error to 

33 all nodes in the network. Ho attempt is made to handle errors in 

34 memory write transfer packets differently from errors detected in | 

35 other ^fpes of packets because once an error is detected in a ^ 
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1 packet, the entire contents of the packet are s^uspect including the 

2 packet type field. 

3 The method of reporting errors is illustrated in Figures 26 

4 amid 27* When a FMC In a cluster detects an error in a packet 

5 received direr the high speed link or detects an erxfor iii a MC bus 

6 transfer, it reports the error to the node'(s) in the cluster by; 

7 1) forcing a parity error on the MC bus and/or 2) directly 

8 interrupting the hode(8)« To force a parity error, the FMC 

9 arbitrates, for the MC bus and when granted access, generates a 

10 transfer where the parity driven on the bus does not natch the 

11 address and data, ^s causes^ the MC port of the each node to 
,12 detect a transfer with bad parity and (hopefully) report it to the 

f'Y " via a parity eirror interrupt. The direct interrupt approach 

V 14 utilizes one or more of the FHC^s eight output interrupt lines. 

15 Khen a IHC in a hub detects an error in a transfer received 

16 over the hub MC bus or detects an error in a packet receiv.ed over 

17 its high speed link, it builds an error packet and transmits it to 

18 the 7MC at the other end of the high speed link. Receipt of the 

19 error packet caiuses the FMC in the cluster to report the error as 
20^ described earlier. Note that lAiether or not the FMC in the cluster 
il receives the packet correctly, an error will be reported. 

22 To assist vith diagnosis of packet transmission or MC bus 

23 transfer errors r the IMC keeps a copy of the last bad packet (if 

24 any) and the last bad MC bus transfer (if any) that it received. 

25 These copies can be afscessed at any time via the. configuration 

26 programming link. 

27 During configuration programming, the error reporting 

28 behavior of a FMC in a cluster is established. The FMC can be 
^ 29 programmed to report errors by generating MC bus parity errors 

30 and/or by generating interrupts. If the interrupt approach is 

* 31 selected, one or more output interrupt lines of the FMC can be 

32 programmed as outputs for error signals. Each of the selected 

33 lines can be further qualified as to vhen the line is pulsed: i) 

34 vhen li bad MCbus transfer is detected (or %7hen an error packet is 

V 35 received indicating that the remote FMC detected a bad transfer) , 
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1 2) when a bad packet is received over the high speed link (or when 

2 an error packet is received indicating that the remote FMG detected 

3 . a bad packet), or 3) when either a bad transfer or a bad packet is 

4 detected* 

5 Although th^e present invention has been shown and described 

6 with reference to preferred embodiments, changes and modifications 

7 are possible to those skilled in the art which do not depart from 

8 the is^irit and contemplation of the inventive concepts taught 

9 herein* Such are deemed to fall within the purview of the 
id invention as claimed* 
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We claim: 

1« A system comprising: 

a first and second set of plurality of nodes; 

a first data bus associated vith and connecting said first sett 
of plurality of nodes; 

a second data bus associated with and connecting said second 
set 6f plurality of nodes; 

ekch node of said sets of. plurality of nodes including a 
processing unit, a miemory, a bus coupled to the processing unit and 
memory, and a sensor means for sensing a write to the memory and 
f Or transiaitting the sensed write on said associated data bus; 

first converter means connected to the first dat& bus for 
cbxiveirting data oh said first data bus to corresponding optical 
signals and received optical signals to corresponding data for 
trainkmission on said first data bus; 

second converter means connected to the second data bus for 
converting data on said second data bus to corresponding optical 
Signals and received optical signals to corresponding data for 
transmission oh said second data bus; and 

fiber optic means for optically transmitting data from one 
converter means to the other converter means « 

2. The systeni of claim 1 wherein each said converter means 
includes conversion means for converting parallel data to serial 
optical signals and vice versa. 

3. The syst^em of claim 1 wherein each said converter means 
includes a FIFO for temporarily storing data. 

4. The system of claim 1 wherein each node further includes I/O 
means for introducing I/O data~ into memory and wherein said sensor 
means responsive to a write to memory of I/O data transmits same on 
said associated data bus* 

5. A system for connecting memory coupled systems, comprising: 
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a plurality of nodes; 

a first data btis connecting a first group of said plurality of 
nodes; 

a secbnd data bus connecting a second group of said plurality 
of nodes; 

first converter means connected to the. first data bus; 

second converter means connected to the second data bus; 

first fiber optic means for ciarrying transmitting data froia 
the first converter means to the second converter means; 

secbnd fiber optic means for carrying transmitted data from 
the second converter means to the first convertefr means; 

the first and second converter means each comprising: 

input leitch means for receiving data from a respective 

data bus; 

hit and translation RAM means connected to the input 
latch means for determining the destination of data received from 
the data bus; 

first micro-interface means connected to the input latch 
mieail^ for controlling the hit and translation RAH means; 

transmission FIFO means connected to the input latch 
means for latching the data received from the data bus; 

error detection means for determining if an error exists 
in the data in the transmission FIFO means; 

first and second transmission latches connected to the 
transmission FIFO means; 

transmitter meems connected to the first and second 
transmission latches for transmitting the data to another converter 
means; 

receiver means for receiving data transmitted from 
ahothier convezrter means; 

first and second receiver latches for latching the 
received data; 

error detection means connected to the first and second 
latch means for checking if an error exists in the received data; 

second micro interface means for testing the received 
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data if the bheck by the error detection means fails; 

receive FIFO means connected to the first and second 
receive latch means for holding the received data after checking by 
the errot detection means; 

outptit latch means for transmitting the received data to 
the respective data bus. 

6. A system as claimed in Claim 5, further comprising first and 
setibnd backup controllers for transmitting data biatveen memory 
coupled systems upon a determination the first and second 
controllers are not working properly. 

7. A system as claimed in Claim 6, wherein said input, output, 
traihsmit aitid receive latches of each of said plurality of 
controllers are able to be accessed In both a parallel and serial 
fashion. 

B. A system ais claimed in claim 1, wherein data is transmitted 
through the optical fiber means in 80 bit data frames. 

9. A miethod comprising the steps of: 

establishing first and second data links; 

establishing first and second sets of nodes, each node 
ihbludlhg a processing unit, a memory, a bus coupled to the 
processing unit and memory and a sensing means for sensing a write 
to memory; 

sensing a write to a memory in one of said nodes of said first 
set; ' 

transmitting said sensed write on said first data link; and 
sensing the sensed write transmitted on said first data link 
and optically transmitting same to a remote point whereupon it is 
transmitted on said second data link to the memory of one of the 
nodes of said second set without intervention of the processing 
unit of said one node of said second set. 
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Id. The method Of claim 9 comprising the further steps of t 

writing I/O data from an I/O source into the memory of a node 

in one of the sets of nodes; 

sensing the written I/O data and transmitting same on the data 

link associated with said node; and 

optically transmitting the written i/0- data to a memory in a 

node of the other sets of nodes via its associated data link. 
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